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13  ABSTRACT 


The  ARP A  computer  network  provides  a  communication  medium  which  allows 
dissimilar  computers  (Hosts)  to  interchange  information.  Each  Host  is 
connected  to  an  Interface  Message  Processor  (IMP),  and  IMPs  are  inter¬ 
connected  by  leased  common  carrier  circuits.-  There  is  frequently  no 
direct  circuit  between  two  communicating  Hosts,  and  the  intermediate 
IMPs  store  and  forward  the  information.  IMPs  regularly  exchange  in¬ 
formation  which  is  used  to  adapt  routing  to  changing  network  conditions 
IMPs  also  report  a  variety  of  parameters  to  a  Network  Control  Center, 
which  coordinates  diagnosis  and  repair  of  malfunctions  -The  Terminal 
IMP  (TIP)  permits  the  direct  attachment  of  63  character-oriented 
terminals.  The  Satellite  IMP  (SIMP)  will  allow  multi-station  use  of  a 
single  earth  satellite  channel.  A  High  Speed  Modular  IMP  (HSMIMP) 
is  under  development;  one  goal  of  this  effort  is  to  increase  IMP 
performance  by  an  order  of  magnitude.  Specialized  mini-Hosts  under 
development  wi.ll  provide  for:  connection  of  remote  batch  terminals; 
simulation  of  a  leased  point-to-point  circuit;  encrypted  Host 
communication. 
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1.  OVERVIEW 

This  Quarterly  Technical  Report,  Number  4,  describes  aspect 
of  our  work  on  the  ARPA  Computer  Network  under  Contract  No. 
F08606-73-C-0027  during  the  fourth  quarter  of  1973.  (Work  per¬ 
formed  from  1969  through  1972  under  Contract  No.  DAHC-15-69-C- 
0179  has  been  reported  in  another  series  of  Quarterly  Technical 
Reports  numbered  1-16.) 

During  the  quarter  we  delivered  one  316  IMP  to  the  Naval 
Air  Station  at  Moffett  Field  (California)  and  three  TIPs  to 
Wright  Patterson  Air  Force  Base  (Ohio),  Rutgers  University  (N.'w 
Jersey),  and  the  Network  Control  Center  at  BBN.  In  conjunction 
with  the  installation  of  the  new  TIP  at  BBN,  the  516  IMP  at  ":BN 
was  moved  from  the  NCC  area  to  the  TENEX  area.  In  addition,  the 
IMP  which  was  shipped  to  MIT’s  Information  Processing  Center 
during  the  third  quarter  was  connected  into  the  network.  The 
Spare  TIP  was  sent  to  replace  the  TIP  at  NOAA/Department  of  Com¬ 
merce  (Colorado)  because  of  hardware  failure  in  their  machine, 
which  was  later  returned  to  Honeywell  for  repair  or  rep  acement . 

During  this  quarter  we  began  field  retrofits  of  the  IMP'S 
Real  Time  Clock.  (The  new  clocks  have  been  installed  ir  all  new 
machines  starting  with  the  Norway  TIP  in  early  June.)  The  new 
Real  Time  Clock  incorporates  a  seven-word  configuration  card  in 
place  of  the  14-bit  IMP  number  card.  This  expansion  will  enable 
the  software  to  adapt  more  easily  to  a  greater  number  of  hard¬ 
ware  configurations. 

By  the  end  of  the  fourth  quarter  there  were  four  Very  Dis¬ 
tant  Hosts  in  the  network.  During  this  last  quarter,  the  VDH 
program  has  been  changed  to  ease  the  introduction  of  a  new 
VDH  site.  The  program  now  determines  the  Host-modem  pair  used 
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by  the  Very  Distant  Host  at  run  time  (at  this  time  only  one  VDH 
Is  allowed  on  an  IMP),  rather  than  requiring  a  specially  tailored 
system  for  each  site. 

The  IMP  program  has  continued  to  undergo  changes  through 
this  quarter.  Many  of  these  changes  are  associated  with  efforts 
to  Improve  the  reliability  and  efficiency  of  packet  routing  in 
the  network.  The  changes  which  have  occurred  during  the  past 
half-year  are  summarized  in  Section  2.  During  our  investigation 
of  routing  techniques,  Dr.  Price  from  the  NPL  spent  a  day  visit¬ 
ing  BBN,  the  main  topic  of  discussion  being  isarithmic  networks. 
These  discussions  indicated  that  isarithmic  networks  have  advan¬ 
tages  over  the  ARPA  Network  flow  control  only  at  times  of  network 
overload. 

A  recent  lockup  of  reassembly  storage  in  the  network  has 
caused  us  to  rethink  the  allocation  mechanism  for  reassembly 
blocks.  A  summary  of  the  problem  and  a  possible  solution  are 
presented  in  Section  3. 

A  survey  of  the  High  Speed  Modular  IMP  (HSMIMP)  and  its 
current  status  is  given  in  Section  4. 

Work  on  the  Satellite  IMP  has  continued  on  several  fronts 
during  this  quarter.  Our  interactions  with  COMSAT  have  con¬ 
tinued  in  an  effort  to  resolve  issues  of  system  design  for  con¬ 
necting  Satellite  IMPs  to  a  broadcast  satellite  channel.  COMSAT 
has  been  particularly  concerned  with  some  of  the  political  issues 
which  must  be  resolved  before  such  a  connection  can  be  realized. 
This  has  resulted  in  COMSAT  obtaining  INTELSAT  approval  of  a 
tariff  for  broadcast  operation.  Some  progress  has  also  been  made 
in  understanding  the  modifications  which  must  be  made  to  the 
SPADE  system  channel  unit .  A  survey  of  the  Satellite  IMP  program 
and  system  is  included  in  Section  5. 
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The  Network  Control  Center  machine  has  begun  expansion  in 
several  directions.  The  machine  itseli  (currently  identical  to 
a  316  IMP  with  no  modem)  is  being  upgraded  to  a  TIP  to  provide 
better  backup  with  other  machines  in  the  NCC.  The  NCC  program 
has  been  modified  so  that  privileged  users  on  BBN-TENEX  or  the 
PDP-1  can  examine  the  contents  of  memory  locations.  This  will 
eventually  ease  putting  network  status  information  on-line,  and 
will  permit  these  Hosts  to  verify  the  memory  of  the  NCC. 

In  an  effort  to  improve  the  reliability  of  the  network  from 
a  TIP  user's  point  of  view,  a  project  will  be  completed  in  early 
January  which  will  test  the  dial-up  modems  in  use  on  network 
TIPs.  When  this  system  is  in  operation,  a  program  on  BBN-TENEX 
will  (at  an  interval  of  perhaps  one  hour)  call  each  dial  -up 
modem  in  the  network.  If  the  connection  is  successful,  the 
program  will  then  send  the  hunt  character,  an  A^CII  "E",  and 
examine  the  response.  An  incorrect  response  will  be  noted,  and 
will  cause  the  NCC  operators  to  be  notified  if  it  persists. 

Finally,  during  this  quarter  BBN  has  continued  its  discus¬ 
sions  of  the  Private  Line  Interface  design  with  ARPA  and  NSA. 
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2.  CHANGES  TO  ROUTING  1 

The  last  half  of  this  year  has  seen  a  number  of  changes  to 
the  routing  program  In  the  IMPs.  These  modifications  can  be 
grouped  under  the  general  heading  of  improvements  in  the  reli¬ 
ability,  the  flexibility,  and  the  efficiency  of  the  propagation 
of  routing  information.  After  discussion  with  ARPA  and  other 
interested  parties,  it  was  decided  to  implement  these  modifica¬ 
tions  before  any  steps  were  taken  toward  high  bandwidth  routing, 
satellite  routing,  or  area  routing;  however,  we  have  made  an 
effort  to  take  our  olans  in  those  areas  into  account  in  designing 
the  new  routing  propagation  techniques. 

Specific  modifications  made  to  date  include  the  following. 
2.1  Efficiency 

The  goal  here  is  to  make  the  propagation  of  routing  data  as 
rapid  as  possible. 

•  The  process  of  changing  the  best  route  to  an  IMP  is 
delayed  for  a  few  seconds.  This  time  allows  the 
adjacent  IMPs  to  learn  of  the  change,  adapt  to  it,  and 
therefore  not  conflict  with  the  changeover.  By 
simulation  and  some  measurement  of  the  network,  we  see 
that  this  technique  speeds  up  the  propagation  of  new 
information  appreciably,  by  1/3  to  1/2. 

•  The  routing  computation  has  been  made  incremental 
and  input-driven,  so  that  the  routing  tables  are 
updated  within  a  few  milliseconds  of  the  arrival  of 
new  routing  data,  rather  than  hundreds  of  milliseconds 
later  on  a  timeout.  Therefore,  the  routing  information 
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in  IMPs  many  hops  away  is  more  up  to  date  by  several 
seconds . 

•  The  routing  tables  have  been  made  smaller  and  faster 
to  manipulate.  There  is  no  longer  a  routing  table  per 
adjacent  IMP,  eliminating  a  costly  multiplicative  factor 
in  storage  costs.  The  routing  tables  are  triDle- 
buffered,  one  for  input,  one  for  output,  and  one  idle. 
This  allows  rapid  table  updates  without  copying,  and 
without  the  need  to  synchronize  with  output.  Again, 

the  propagation  of  routing  is  speeded  up  by  providing 
the  output  process  with  the  most  up-to-date  table 
possible . 

•  The  output  of  routing  messages  is  triggered  by  the 
arrival  of  new  routing  data,  rather  than  being  strictly 
synchronous.  This  completes  the  system  for  rapid 
propagation  of  routing  changes. 

2.2  Flexibility 

The  goal  here  Is  to  extend  the  operation  of  the  routing 
program  to  different  line  speeds. 

•  The  IMF  program  measures  the  bandwidth  of  the  circuits 
to  which  it  is  connected,  and  the  excess  capacity 
available  on  each  circuit. 

•  Routing  messages  are  sent  at  a  rate  proportional  to 
the  bandwidth  of  a  circuit. 

•  Routing  messages  are  also  sent  more  often  when  a  circuit 
is  lightly  loaded,  to  improve  the  speed  with  vnich 
alternative  paths  can  be  recognized.  The  overhead  due 
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to  routing  messages  therefore  ranges  between  about  3% 
and  15?,  depending  on  loading,  for  any  line  speed  in  the 
range  5  Kbs-25C  Kbs. 

•  At  the  same  time,  the  criteria  for  declaring  circuits 
usable  have  been  made  variable  with  circuit  bandwidth, 
allowing  more  errors  on  slower  lines  before  declaring 
them  unusable. 

2.3  Reliability 

The  goal  here  is  to  localize  and  minimize  the  effects  of 
any  hardware  or  software  failure  in  the  routing  process. 

•  Routing  messages  carry  a  checksum  generated  in  software 
at  the  time  the  message  is  built. 

•  The  checksum  is  verified  at  the  time  the  message  is 
sent  to  catch  intra-IMP  failures  such  as  processor  or 
memory  errors,  software  bugs  wtp  ch  change  the  data,  and 
so  on. 

•  The  checksum  is  verified  at  the  time  the  message  is 
received  to  catch  inter-IMP  failures  such  as  bus  transfer 
or  modem  interface  errors,  failures  in  the  memory  at  the 
receiver,  and  so  on. 

•  Errors  in  both  categories  have  been  detected.  When  an 
error  is  detected,  the  routing  message  is  discarded, 
preventing  the  spread  of  any  erroneous  routing  data 
through  the  network,  and  a  copy  is  sent  to  the  NCC  for 
diagnosis . 
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•  The  routing  directory  held  at  each  IMP  also  carries 
a  checksum,  which  is  incrementally  modified  whenever 
an  entry  changes,  and  periodically  checked.  Errors 
have  been  detected  here  also,  and  result  in  a  program 
reload. 

•  In  addition  to  checksums  on  routing  data,  the  IMP 
maintains  checksums  on  all  routing  programs.  The 
input  routing  processing  code  is  checksummed  before 
it  is  executed;  the  same  is  true  of  the  output 
routing  processing  code  and  the  periodic  routing 
processing  code.  Errors  in  the  program  code  have  been 
detected  by  this  means,  and  result  in  immediate  reload 
of  the  program  before  the  erroneous  program  is 
executed  even  once. 

•  Tne  goal  of  these  measures  has,  in  large  part,  been 
achieved,  since  several  failures  have  been  detected 
and  corrected  while  local,  preventing  network-wide 
repercussions . 

In  summary,  the  first  phase  of  the  modifications  to  the 
routing  program  is  largely  finished.  Some  refinements  and 
extensions  to  the  techniques  described  may  be  implemented  in 
the  future,  but  the  main  thrust  or  our  work  will  now  be  in 
the  remaining  areas  of  routing  development  named  above — high 
bandwidth  routing,  satellite  routing,  and  area  routing. 
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3.  A  REASSEMBLY  LOCKUP 


Near  the  end  of  the  fourth  quarter,  a  new  kind  of  reassembly 
lockup*  occurred  around  the  UCLA  IMP,  precipitating  a  n^w  look  at 
the  way  the  IMP  system  handles  reassembly.  The  NMC  had  turned  cn 
cumulacive  statistics  and  snapshots  statistics  all  around  the  net¬ 
work.  The  lockup  happened  when  a  message  arrived  at  UCLA,  which 
had  reassembly  packet  buffers  allocated  for  it,  but  no  reassembly 
blocks  were  free.  A  reassembly  block  is  a  piece  of  storage  used 
in  the  actual  process  of  putting  the  packets  back  together.  Both 
a  reassembly  block  and  up  to  8  packet  buffers  are  needed  to  re¬ 
assemble  a  mesaag°.  The  IMP  reserves  the  packet  buffers  in  ad" 
vance,  and  assumes  that  there  is  always  a  free  reassembly  block 
available  The  lockup  pointed  out  an  oversight  in  the  IMP  pro¬ 
gram:  not  enough  reassembly  brooks  had  been  assigned  to  cover 

all  situations. 


The  lockup  lasted  fo-  a  few  hours,  while  the  NCC  staff 
located  the  problem,  and  normal  operations  resumed  when  they  con¬ 
tacted  the  NMC  and  requested  that  the  NMC  stop  the  experiment. 
Subsequently,  the  IMP  program  has  been  changed  to  allow  the  NMC 
to  continue  its  experime  .ts  without  any  possibility  of  netv'ork 
lockup.  Having  experienced  this  problem  and  developed  a  solution, 
we  reexamined  the  reassembly  processing  in  the  IMP. 


*See  QTR  number  13  in  the  previous  QTR  series  for  a  discussion 
of  a  type  of  reassembly  lockup  whl^h  was  discovered  and  fixed. 
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How  many  reassembly  blocks  are  needed?  The  worst-case 
analysis  is  as  follows,  given  reassembly  storage  for  N  packets 


1)  A  large  number  of  requests  for  allocation  arrive  at  once. 

2)  N/8  allocates  are  sent,  each  good  for  an  8-packet  message, 

3)  These  messages  turn  out  to  be  2-packet  messages;  the 
second  packets  arrive  before  the  first;  the  remaining 

6  packet  buffers  for  each  message  are  declared  free  for 
other  uses. 

(N  -  2*  N/8)  /8  =  3N/32  allocates  are  sent. 

Same  as  3. 

(N  -  2*  3N/32)  /8  =  13N/128  allocates  are  sent. 


4) 

5) 

6) 


And  so  on;  eventually,  all  storage  is  allocated  and  N/2  reassembly 
blocks  are  in  use. 


3.1  A  Simple-Minded  Solution 


Sine  '  the  core  retrofit  (described  in  QTR  Number  1,  April 
1973)  has  given  us  approximately  9  x  8  =  72  packet  buffers  for 
reassembly,  there  could  conceivably  be  36  messages  being  re¬ 
assembled  at  once.  One  way  to  eliminate  lockups  would  be  to 
assign  36  reassembly  blocks  as  part  of  the  IMF’s  permanent 
storage.  But  each  block  is  currently  12  words;  this  requirement 
of  36  x  12  =  132  words  is  excessive,  considering  that  it  is  rare 
to  be  reassembling  more  than  2  or  3  messages  at  once.  The  whole 
network  went  for  a  year  before  an  IMP  had  to  reassemble  more 
than  8  messages  at  once. 


3.2  The  Current  Solution 


In  response  to  the  lockup,  we  changed  the  procedure  at  the 
destination  IMP  to  look  like  this: 
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11  Get  request  for  allocation. 

2)  Wait  till  storage  is  free  for  B  packets. 

2a)  Wait  till  reassembly  block  count  shows  a  free  block.* 

3)  Send  allocate. 

3a)  Increment  reassembly  block  count.* 

*0  Receive  some  packet  of  the  message  (not  necessarily 
the  first). 

Find  a  free  reassembly  block  (should  always  be  possible) 
the  format  is: 

-  chain  pointer 

-  message  number 

-  source  IMP  number 

-  number  of  rackets  in  this  message 

-  number  of  packets  received  so  far 

-  8  packet  pointers 

Take  the  block  from  the  free  list  and  put  it  on  the 
active  list. 

Set  up  the  message  number,  IMP  number. 

5)  For  each  packet  received: 

Set  up  the  packet  pointer  (duplicate  packets  detected 
here) . 

Increment  the  received  count. 

6)  When  the  last  packet  is  received: 

Set  up  the  number  of  packets  in  this  message. 

7)  When  all  packets  have  been  received: 

Wait  till  message  number  is  next  in  sequence,  then  chain 
packets  together,  put  on  Host  queue,  and  free  reassembly 
block. 

7a)  Decrement  reassembly  block  count.* 

*These  are  the  program  changes. 
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3.3  Reassembly  Reexamined 

A  natural  direction  for  cnange  in  the  future  is  to  tie  the 
reassembly  block  to  the  allocation  of  packer  storage.  The  re¬ 
mainder  of  this  section  will  discuss  a  possible  change  along  this 
line . 


At  the  time  the  allocation  is  sent  (stop  3  above),  the  IMP 
can  wait  until  it  can  take  a  reassembly  block.  Then  it  can  add 
the  block  to  the  active  list,  mark  the  IMP  number  and  note  the 
number  of  packets  received  as  -1,  to  denote  an  empty  block.  Not 
only  does  this  ensure  that  a  reassembly  block  exists  for  every 
message,  simplifying  the  first-packet  processing,  but  it  has  an 
important  additional  benefit.  This  gives  us  a  data  structure  to 
record  allocates  that  have  been  sent  and  are  unused.  No  such 
structure  exists  currently,  so  allocates  sent  off  to  an  IMP  which 
dies  or  malfunctions  are  not  explicitly  cleaned  up.  (There  is  an 
idle  timer  which  causes  all  outstanding  allocates  to  be  voided 
after  2  minutes  of  inactivity,  but  this  mechanism  is  not  very 
satisfactory . ) 

With  this  scheme,  when  a  packet  of  a  multipacket  message 
arrives,  the  program  will  search  the  reassembly  queue  for  that 
message  number  and  IMP  number.  If  a  match  exists,  the  message  is 
already  being  reassembled.  If  not,  a  second  search  is  made  for  a 
block  for  that  IMP  which  is  empty  (number  of  packets  received  = 
-1).  There  should  always  be  a  block  of  one  type  or  the  other. 

Additionally,  when  an  IMP  goes  down,  all  the  other  IMPs 
in  the  net  can  clean  up  their  reassembly  queues  in  a  uniform 
fashion,  freeing  the  reassembly  space  and  packets  from  blocks 
in  use,  the  allocation  from  empty  blocks,  and  the  blocks  them¬ 
selves  in  either  case.  Also,  the  processing  of  give-backs  can 
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be  made  a  little  tighter,  since  a  reassembly  block  should  be  set 
up  for  every  give-back  received. 

3.4  An  Extension  with  Message  Numbers  by  Host 

We  are  planning  to  change  the  IMP  to  assign  message  numbers 
on  a  per  Host  basis,  and  have  been  planning  to  use  one  message 
number  for  both  the  8-packet  request  and  the  message.  This  is 
possible  since  on  a  per  Host  basis,  no  other  traffic  can  flew 
between  the  request  and  the  message.  When  we  do  this,  the 
process  which  acquires  the  reassembly  block  before  sending  off 
the  allocate  can,  in  addition  to  setting  up  the  IMP  number,  set 
up  the  message  number  of  the  reassembly  block.  This  eliminates 
the  need  for  a  double  search  of  the  reassembly  queue  for  the  first 
packet.  The  search  is  still  necessary  in  the  case  that  the  mes¬ 
sage  is  using  a  piggybacked  allocate  rather  than  an  explicitly 
requested  one. 

3.5  An  Extension  with  Message  Length  Given  by  Host 

We  are  also  planning  to  add  a  message  length  to  the  Host-IMP 
leader,  to  allow  the  Host  to  tell  the  IMP  how  long  its  message  is. 
This  information  can  be  used  to  request  an  allocation  of  exactly 
as  many  packets  as  are  necessary.  Then  when  the  reassembly  block 
is  acquired  prior  to  sending  the  allocate,  the  exact  size  of  the 
allocation  can  be  saved  in  the  "number  of  packets  in  this  message" 
field.  Again,  piggybacked  allocates  are  different,  and  will 
always  refer  to  8  buffers,  but  an  improvement  is  possible  here 
also.  Currently,  each  packet  has  3  bits  for  packet  number  and 
1  bit  for  last  packet  or  not.  When  we  Know  message  size  ahead  of 
time,  the  packet  header  can  contain  3  bits  of  packet  number  and 
3  bits  of  "total  packets  in  this  message".  Then  when  any  packet 
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of  a  message  is  received,  the  surplus  between  the  number  o''  packets 
allocated  and  the  number  of  oackets  in  the  message  can  be  freed. 

3.6  Summary 

The  reassembly  processing  would  then  look  like  this  (compare 
with  section  3.2): 

1)  Get  request  for  allocation  of  N  packets  (or  have  RFNM 
on  which  to  piggyback  an  allocation  of  N  =  8). 

2)  Wait  for  N  packet  buffers  to  be  free;  claim  them. 

3)  Wait  for  a  reassembly  block  to  be  free;  when  it  is: 

-  take  it  off  the  free  reassembly  block  list 

-  set  up  the  IMP  number,  the  Host  numbers 

-  set  up  the  message  number  (if  not  a  piggyback 
allocation) 

-  set  the  total  packet  count  to  N 

-  set  the  packets  so  far  to  -1 

-  put  the  block  on  the  active  reassembly  block  list. 

*0  When  a  packet  arrives: 

-  search  the  active  reassembly  block  3 1st  for  that 
IMP/Host/message  number. 

If  found: 

-  set  up  packet  pointer  (discard  if  duplicate) 

-  set  up  total  packet  count  according  to  packet 
header,  discard  any  difference  between  this 
and  previous  contents  as  free  space 

-  Increment  count  of  received  packets 

•-  done  if  received  packet  count  =  total  packet 


count . 
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If  not  found: 

-  search  for  an  empty  block  for  this  IMP 

-  set  up  message  number  and  do  above  processing. 

Mote  that  we  have  eliminated  any  special  processing  for 
the  last  packet,  and  almost  all  special  p  ’oceosing  for  the 
first  packet.  As  a  final  thought,  even  ",his  processing  could 
be  avoided.  We  plan  to  have  at  least  2  bits  per  message  indi¬ 
cating 

-  request  processed 

-  message  processed 

and  we  could  define  a  state  meaning  "message  being  processed  — 
at  least  one  packet  received"  which  would  differentiate  the 
case  of  empty  arid  non-empty  reassembly  blocks: 


nothing  received  for  this 
message 

request  received 
message  being  processed 
message  completely  processed 


Request  Message 
Legal?  Legal? 


Y  (piggyback 

al locate) 

Y 


For  single  packet  messages,  state  10  is  meaningless.  We  might 
want  to  do  something  similar  to  this  approach  for  single  packet 
messages,  say  claim  a  packet  buffer  and  put  the  allocate  in  it, 
in  order  to  keep  track  of  outstanding  1-packet  allocates.  Then 
we  could  garbage-collect  them  in  the  event  that  the  source  IMP 
goes  down. 
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4.  HIGH  SPEED  MODULAR  IMP 

Descriptions  in  previous  Quarterly  Technical  Reports  of  this 
series  have  concentrated  on  recent  developments.  In  this  section, 
however,  we  will  survey  the  system  as  we  now  see  it.  The  last 
such  survey  can  be  found  in  Number  14  of  the  previous  series  of 
Quarterly  Technical  Reports  mentioned  in  the  Overview.  Another 
r,ecent  description  which  provides  more  insight  into  the  software 
is  found  in  [1], 


The  High  Speed  Modular  IMP  (HSMIMP)  is  designed  to  provide 
an  order  of  magnitude  increase  over  the  throughput  of  the  DDP-516 
IMP.  The  architecture  is  highly  modular  and  allows  for  IMPs  of 
greater  or  lesser  processing  power  than  the  present  5l6/3l6-based 
IMPs,  as  well  as  for  many  more  and  more  varied  phone  line  and 
Host  interfaces.  The  architecture  also  provides  for  highly 
reliable  configurations.  The  hardware  consists  of  busses  joined 
together  by  special  bus  couplers  of  our  design.  Figure  4-1  shows 
the  structure  of  the  prototype  system.  There  are  processor 
busses  each  of  which  contains  two  processors,  each  in  turn  with 
its  own  "private1'  4k  memory  to  store  frequently  run  code,.  The 
more  processor  busses,  the  greater  the  system  processing  power. 
There  are  memory  busses  to  house  the  segments  of  multi  orted 
"common"  memory  —  the  more  memory  busses,  the  more  memory  ports. 
Finally,  there  are  I/O  busses  which  house  device  and  line  con¬ 
trollers  as  well  as  a  special  (priority  ordered)  task  disburser 
(the  PID)  which  replaces  the  traditional  priority  interrupt  sys¬ 
tem.  The  latter  allows  equality  among  the  processors  sc  that  if 
some  fail  the  rest  can  continue  to  run  all  system  tasks,  albeit 
at  reduced  capacity. 
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4.1  design  Issues 

In  this  section  we  describe  features  we  Y  „ve  designed  into 
the  system,  some  of  the  more  interesting  of  which  relate  to  re¬ 
liability. 

4.1.1  Addressing  and  Locking 

The  Lockheed  SUE,  with  a  15-bit  address,  can  directly 
address  only  up  to  32K  words.  8K  of  a  processor's  address  space 
are  used  for  direct  references  to  its  private  memory.  (Although 
we  expect  to  use  only  4K,  8K  has  been  set  aside  to  allow  for 
growth.)  Another  8K  is  used  principally  for  addressing  system 
I/O  (on  the  up  to  four  I/O  busses).  We  assign  8  addresses  to 
each  I/O  device  for  pointers  and  status  and  control  registers; 
960  devices  can  be  accommodated  in  all. 

ItK  of  each  processor's  address  space  is  mapped  through  the 
bus  couplers  to  common  memory.  At  the  processor  end  of  each 
coupler  (BCP)  are  four  program-settable  map  registers  for  each 
possible  processor  on  the  bus.  (We  expect  to  use  only  two  pro¬ 
cessors  per  bus  but  up  to  four  are  permitted.)  These  map  reg¬ 
isters  expand  a  15-bit  address  to  a  19-bit  system  address  on  the 
memory  busses.  By  use  of  the  maps,  each  processor  can  thus 
access,  at  any  one  time,  four  4K  pages  in  system  address  space. 
Read  accesses  through  a  particular  one  of  these  windows  are 
turned  by  the  coupler  into  read-clear  operations,  thereby  pro¬ 
viding  the  indivisible  test-and-modify  operation  required  for 
program  interlocking  in  a  multi-processor.  (The  process  r  it¬ 
self  presently  lacks  such  an  instruction.) 

The  coupler  paths  that  connect  processor  busses  into  memory 
and  I/O  busses  have  program  settable  enabling  switches  at  their 
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far  (memory  and  I/O)  ends,  thus  permitting  processors  to  be  jut 
in  and  out  of  the  system.  To  allow  processors  to  access  one 
another  and  to  permit  reloading  as  discussed  below,  we  have  pro¬ 
vided  reverse  paths  in  the  processor  to  I/O  couplers  which  also 
have  enabling  switches.  Normally  the  forward  paths  to  memory 
and  I/O  are  turned  cn  and  the  backward  paths  are  shut  off.  Since 
these  paths  represent  a  hazard  whereby  a  "sick"  processor  or 
device  could  damage  healthy  processors,  we  have  arranged  that 
only  by  storing  a  password  at  the  proper  address  can  a  switch  be 
changed.  This  greatly  reduces  the  probability  that  a  berserk 
processor  painting  memory  will  affect  the  path.  A  processor  can 
neither  enable  nor  disable  its  own  access  paths  but  one  proces¬ 
sor,  deciding  that  another  is  sick  and  should  be  eliminated  from 
the  system,  can  amputate  the  bus  of  the  offending  processor.  It 
can  be  similarly  reinstated  later. 

The  logic  upon  which  amputation  decisions  are  based  is  not 
yet  fully  understood  and  will  be  worked  out  as  experience  grows. 
We  expect  to  require  all  processors  to  execute  periodic  healthi¬ 
ness-proving  tasks.  A  regular  system  task,  performed  by  any 
free  processor,  verifies  that  all  processors  have  passed  their 
tests  and  amputates  any  unhealthy  one(s).  Protective  embellish¬ 
ments  easily  suggest  themselves  and  we  expect  to  do  what  seems 
necessary . 

4.1.2  Discovery 

The  operational  program  implements  the  IMP  algorithm  with 
whatever  hardware  is  working  at  a  particular  site  at  a  given 
time.  The  program  discovers  the  hardware  configuration  as  fol¬ 
lows:  Memory  is  found  by  trying  to  access  it;  a  failure  inter¬ 

rupt  results  if  Memory  is  not  there.  Processors  are  found  by 
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accessing  a  register  whose  response  indicates  if  the  processor 
is  absent,  running  or  halted.  I/O  Devices  are  found  by  reading 
the  1st  word  of  every  possible  device  in  I/O  space  —  a  failure 
interrupt  means  no  Device,  a  response  returns  a  unique  l6-bit 
device  type.  Any  parameters  needed  to  run  the  devices  are 
available  as  status  words  in  the  8-word  block.  It  is  somewhat 
hax’der  to  find  where  the  bus  boundaries  are,  but  they  too  can 
be  found  by  searching  for  the  bus  coupler  disable  switches.  In 
the  event  that  there  is  some  property  we  cannot  otherwise  dis¬ 
cover,  we  have  set  aside  3  registers  (associated  with  the  Real 
Time  Clock)  to  hold  this  information.  For  example,  the  IMP  num¬ 
ber  (used  for  network  routing)  is  contained  in  8  bits  of  these 
registers . 

The  Discovery  logic  is  not  an  initialization  phase;  rather 
the  program  periodically  runs  through  the  Discovery  logic  and 
reconfigures  whenever  a  change  occurs.  It  thus  automatically 
adapts  the  IMP  algorithm  not  only  to  the  wide  variety  of  possible 
configurations  but  also  to  those  which  contain  broken  components. 

4.1.3  Parity 

At  present  the  memories  we  are  using  do  not  store  parity; 
however,  we  have  built  Into  our  system  design  (and  into  the  hard¬ 
ware)  mechanisms  to  incorporate  parity.  These  mechanisms  have 
been  tested  with  prototype  parity  memory  and  we  have  recently 
ordered  parity  memories  for  our  production  machines.  We  use  a 
novel  parity  computation  based  not  only  upon  the  contents  of  a 
word  but  also  on  its  address.  The  scheme  also  detects  both  "all 
ones"  and  "all  zeros"  failures.  For  writes  to  common  memory, 
parity  is  computed  at  the  processor  end  and  fed,  via  the  coupler, 
to  the  memory  where  It  is  stored  with  the  word.  Reads  from 
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memory  fetch  this  stored  parity,  which  is  compared  to  a  recom¬ 
puted  parity  at  the  processor  end  of  the  coupler,  thus  checking 
both  the  memory  and  coupler  paths  in  both  directions.  For  units 
on  the  I/O  bus,  in  order  to  check  the  coupler  paths,  a  special 
card  computes  and  transmits  parity  for  all  words  being  read  from 
the  I/O  bus  by  the  processors  and  checks  parity  on  all  words 
arriving  from  processor  busses. 

4.1.4  Reloading 

At  present  we  use  paper  tape  to  load  the  system.  The 
operator  starts  a  processor  which,  from  tape,  loads  its  own  pri¬ 
vate  memory,  its  map  registers  and  thereby  any  or  all  of  common 
memory.  It  also  loads,  using  backward  coupling,  the  private 
memories  on  all  ether  processor  busses  in  the  system.  After  the 
memory  has  been  loaded,  a  startup  procedure  is  executed  which 
finally  turns  on  the  other  processors. 

Since  all  crucial  switches,  parameters,  registers  and  con¬ 
trol  flip  flops  have  been  made  addressable  by  reads  and  writes, 
loading  the  system  and  starting  it  up  can  be  done  by  externally 
force  feeding  it  with  the  right  set  of  addresses  and  data.  Al¬ 
though  we  presently  use  paper  tape  in  conjunction  with  a  boot¬ 
strap  ROM  executed  by  a  processor  for  this  purpose,  we  are  plan¬ 
ning  to  construct  a  means  whereby  the  system  can  be  force  fed 
directly  from  the  network.  I  he  mechanism  for  this  is  a  device 
on  the  I/O  bus  which  monitors  phone  lines  from  adjacent  IMPs 
looking  for  a  special  format  which  signals  arrival  of  reload 
information.  The  card  t  en  performs  the  reload  by  executing 
store  type  bus  cycles  usi  T  the  reload  data. 

This  sort  of  operation,  which  looks  forward  to  elimination 
of  paper  tape,  switches,  and  other  operator  dependent  functions. 
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is  appropriate  to  the  IMP's  job.  If  a  running  system  fails,  as 
viewed  from  the  net,  the  first  step  is  to  send  it  a  regular  "for 
IMF"  message  which  causes  a  standard  system  restart  to  be  at¬ 
tempted.  If  that  seems  not  to  work,  the  next  step  is  to  send 
another  regular  message  trying  to  activate  the  reloaa-from- the- 
net  code  in  hopes  that  it  is  still  intact.  Only  if  that  fails 
would  one  attempt  to  force  a  full  restart  from  scratch,  in  which 
case  the  special  card  described  above  is  called  into  play.  The 
first  data  sent  halts  the  processors  in  order  to  stop  any  inter¬ 
fering  activity.  Then  the  reload-from-the-net  code  is  refreshed 
and  finally  a  processor  restarted  running  that  code  which  then 
completes  reload  via  the  normal  packet  mechanism. 

4.1.5  Mechanical  Modularity 

We  have  settled  on  a  modular  mechanical  structure  well 
matched  to  the  modular  logical  structure  of  the  system.  This 
structure  is  Important  in  that  it  allows  easy  construction  of 
systems  of  varied  size  and  permits  repair  of  parts  of  a  system 
while  the  rest  of  it  continues  to  operate.  The  basic  unit  is  a 
cooling  module  which  houses  either  1)  a  16-slot  bus  complete 
with  its  own  power  supply,  2)  a  24-slot  bus  without  power,  or 
3)  a  power  supply  for  such  a  24-slot  bus.  These  units,  each 
with  its  own  set  of  fans,  sit  on  rails  in  a  vertical  tier  in  a 
rack,  five  of  them  filling  a  standard  height  rack.  (The  14- 
processor  system  requires  three  racks.)  Figure  4-2  shows  how  the 
cooling  modules  stack.  Air  flow  is  from  back  to  front  so  that 
racks  placed  beside  one  another  do  not  directly  heat  each  other. 

A  tilted  pan  at  the  bottom  of  each  module  separates  the  air  flow 
between  stacked  modules,  thus  eliminating  chimney  effects.  Cards 
plug  in  from  the  front  and  all  device  and  coupler  cables  also 
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connect  on  that  side.  An  entire  unit  can  be  removed  to  the  rear 
for  repair  or  replacement  of  the  bus,  fans,  etc.  —  all  without 
disturbing  operation  of  the  remainder  of  the  system. 

4.2  The  Test  Program 

The  primary  design  objective  of  the  test  program  is  to 
exercise  all  of  the  hardware  as  intensively  and  extensively  as 
possible,  detecting  all  failures  and  reporting  them  precisely 
and  comprehensibly.  Extensive  testing  implies  a  vid'  variety  of 
test  modules;  intensive  testing  implies  permitting  the  entire 
computational  power  of  the  system  to  be  focused  on  individual 
components  at  times.  These  objectives  led  to  the  selection  of  a 
system  based  on  processes,  analogous  to  a  time-shared  system's 
jobs.  Proceoses  are  not  tied  to  processors;  a  given  process 
will  switch  rapidly  from  one  processor  to  another.  Nor  is  a 
process  in  general  tied  to  a  specific  copy  of  code;  like  time- 
shared  jobs,  processes  share  a  single  copy  of  sections  of  pure 
procedure . 

There  are  four  types  of  processes:  the  "system"  processes, 
including  the  clock,  timeout,  and  type-out  processes;  the  device¬ 
specific  processes,  which  are  tied  to  particular  I/O  devices,  two 
processes  per  device;  the  "GART"  (Get  A  Random  Test)  processes, 
which  select  a  test  at  random  from  a  table  of  tests  to  be  per¬ 
formed;  and  a  dummy  process,  whose  sole  purpose  is  to  assure 
that  there  is  always  a  runnable  process. 

Each  GART  test  Is  designed  to  test  a  particular  element  or 
feature  of  the  system.  These  range  from  standard  processor  and 
memory  tests  (the  latter  are  also  useful  for  checking  bus  coup¬ 
lers)  to  exercising  the  various  bus  coupler  switches  and  maps. 

The  I/O  devices  are  kept  busy  by  circulating  various  data  through 
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them.  We  are  currently  In  the  process  of  making  the  test  pro¬ 
gram  more  robust  so  that  it  can  survive  transient  problems,  re¬ 
port  on  them  and  continue. 

4.3  Where  We  Stand 

Although  the  system  uses  Lockheed  SUE  processors,  cusses, 
memories,  etc.,  we  have  so  far  designed  and  built  nine  BBN  card 
types  for  the  system:  three  coupler  cards  for  each  of  the  three 
bus  types,  a  full-duplex  memory  channel  card,  a  Host  interface 
card  (which  operates  at  speeds  up  to  1.5  megabit),  transmit  and 
receive  modem  cards,  the  pseudo-interrupt  card  and  a  clock  card. 
These  designs  are  virtually  all  finalized  and  many  are  in  pro¬ 
duction  (printed  circuit  or  similar)  form. 

We  are  presently  finishing  the  design  of  two  other  cards: 
the  first  of  these  Is  the  parity  checking  card  for  the  I/O  bus 
described  above  under  the  discussion  of  parity.  The  second  is 
a  checksum/block-transfer  card  which  flows  a  block  of  memory 
through  itself  computing  a  checksum  as  it  goes.  This  is  used 
to  checksum  critical  code  from  time  to  time  [2],  to  compute 
checksums  for  network  end-to-end  checking  of  messages,  and  other 
useful  checking  purposes.  A  transfer  mode  can  be  enabled  so 
that  it  can  also  be  used  to  move  blocks  of  information  about  in 
memory  (checksumming  as  it  goes  if  desired).  In  addition  we  are 
presently  embarking  on  modifications  to  the  modem  transmit  and 
receive  cards  which  will  allow  them  to  deal  with  1.5  megabit 
lines  and  design  of  the  special  interface  which  monitors  incoming 
Inter-IMP  lines  watching  for  reload  information  as  described 
above . 

At  present  we  are  running  several  systems.  Two  small  sys¬ 
tems  are  being  used  for  testing  and  debugging  of  the  IMP  program. 


—  4 


.  1  3 

t-  J 


) 


j 

i 


l 


j 


24 


Report  No.  2717 


Bolt  Beranek  and  Newman  Inc. 


These  are  sometimes  run  as  separate  single  bus  IMP  systems  which 
ai'e  connected  together  with  our  prototype  516  IMP  Into  a  three- 
node  network.  At  other  times  the  two  busses  are  combined  into 
a  single  system  using  a  bus  coupler.  In  this  case  one  bus  is 
used  as  a  dual  processor  bus  and  the  other  as  a  combined  memory 
and  I/O  bus.  This  system  then  works  with  tne  516  IMP  to  form  a 
2-node  net. 

The  growing  prototype  l^-processor  system  presently  consists 
of  three  dual  processor  busses,  two  memory  busses  and  one  I/O  bus 
(although  a  larger  system  with  six  of  the  seven  processor  busses 
has  successfully  run  the  test  program) .  The  system  has  grown 
gradually  and  it  now  operates  with  reasonable  reliability  under 
stress  (shaking  off  cables,  margining  power  supplies,  shuffling 
of  cards,  etc-;.  By  mid-197^  we  hope  to  have  two  production 
copies  of  the  large  prototype  working  in  the  network.  During 
197*1  we  plan  also  to  design  satellite  modem  interface  cards  and 
to  produce  and  deliver  three  moderate  sized  systems  with  satel¬ 
lite  capability.* 

The  basic  IMP  system  program  is  up  and  running  in  multi¬ 
processor  form,  that  is,  with  processors  picking  tasks  up  via 
the  pseudointe.rrupt  system  and  using  locks  to  prevent  interfering 
accesses  to  resources.  It  has  been  run  on  a  five-processor  sys¬ 
tem  for  short  periods  although  most  debugging  has  been  on  a  two- 
processon  system.  The  inner  parts  of  the  system,  store  and 
forward ,  Host,  task,  etc.,  seem  solid.  The  work  that  remains  is 
in  implementing  the  system  maintenance,  monitoring,  and  debugging 
functions  (i.e.,  system  DDT,  periodic  status  reports,  etc.).  Thi 


*See  section  5  of  this  QTR. 
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coding  is  about  half  done  and  needs  finishing  as  well  as  debugging. 
The  network  error  recovery  code  is  ready  for  debugging.  The  spe¬ 
cial  reliability  code  which  keeps  the  system  up  when  parts  of  the 
hardware  fail  is  being  designed. 

At  the  other  end  of  the  performance  spectrum,  the  small  IMP 
is  built  on  a  single  logical  bus  (consisting  of  two  separate 
physical  tusses  connected  by  an  extender)  which  combines  memory, 
processor  and  I/O.  This  system  embodies  none  of  the  special 
reliability  stemming  from  multiple  hardware  copies  but  is  the 
least  expensive  version  available.  Small  reliable  systems  are 
another  matter  and  require,  in  general,  doubling  the  system  to 
provide  complete  redundancy  of  parts  to  allow  for  any  single 
failure.  Such  systems  may  prove  to  be  one  of  the  more  significant 
outgrowths  of  this  development  effort. 
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5.  THE  SATELLITE  IMP 

The  communication  circuits  connecting  the  IMPs  in  the  ARPA 
Network  have  typically  been  50Kbs  ground  links,  although  9*oKbs 
and  230.4Kbs  links  are  also  in  use.  In  the  past  year,  two  satel¬ 
lite  links  have  been  added  to  the  network,  one  a  50Kbs  link  from 
California  to  Hawaii  and  the  other  a  9.6Kbs  link  from  Washington, 
D.C.  to  Norway.  In  both  cases  these  satellite  links  are  conven¬ 
tional  point-to-point  links. 

The  likely  introduction  of  further  satei'lte  links  into  the 
ARPA  Network  as  it  expands  overseas,  and  the  „oming  possibility 
of  domestic  satellite  communication,  have  led  to  the  development 
of  a  new  variant  of  the  IMP  technology  called  the  Satellite  IMP. 
The  Satellite  IMP  permits  several  network  nodes  to  share  a  single 
satellite  channel,  enabling  the  nodes  to  statistically  average 
their  total  load  (at  the  satellite)  rather  than  requiring  each 
node-pair  to  average  their  traffic  independently  [3].  The  manner 
in  which  the  Satellite  IMP  provides  such  channel  sharing  is  based 
on  the  concept  of  packet  broadcast,  first  used  in  the  ALOHA 
System  [4], 

When  using  a  packet  broadcast  protocol,  all  nodes  sharing 
the  channel  transmit  discrete  packets  of  data  on  the  same  trans¬ 
mission  frequency.  All  nodes  sharing  the  channel  also  receive 
on  the  same  receive  frequency,  picking  out  packets  addressed  to 
themselves  and  discarding  packets  addressed  to  others. 

The  original  ALOHA  or  "randon"  ALOHA  system  permits  nodes 
to  transmit  their  packets  in  a  completely  uncoordinated  way. 

Thus,  there  is  a  substantial  possibility  of  transmission  conflict 
(and  consequent  destruction  of  the  conflicting  packets)  with  a 
frequent  need  for  retransmission  of  packets.  The  retransmission 
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must  somehow  be  randomized  to  avoid  repeated  conflict.  As  shown 
in  [4],  a  channel  operating  in  such  a  manner  has  an  effective 
throughput  capacity  of  18$. 

Roberts,  in  [5],  pointed  out  that  if  an  ALOHA  channel  were 
divided  into  slots  (each  able  to  hold  a  packet)  and  if  the  nodes 
modified  their  behavior  to  transmit  packets  so  that  the  leading 
edge  of  the  packet  always  coincides  with  the  leading  edge  of  a 
slot,  even  though  the  nodes  remain  free  to  transmit  into  a  slot 
without  regard  for  the  transmission  of  other  nodes,  the  effective 
channel  capacity  is  doubled  (to  26$).  A  channel  operated  in  this 
manner  is  called  a  "slotted  ALOHA"  channel. 

The  Satellite  .IMP  represents  a  practical  application  of  the 
large  body  of  theoretical  knowledge  about  the  operation  of  packet 
broadcast  channels  that  has  developed  since  the  ALOHA  System,  and 
has  been  reported  in  [3,  5,  6,  7,  8,  9,  10].  The  Satellite  IMP 
is  a  conventional  IMP  with  several  additions  and  modifications 
both  in  hardware  and  software . 

5.1  Hardware 

The  first  hardware  addition  is  of  memory  sufficient  to  buf¬ 
fer  all  the  transmitted  packets  which  can  be  awaiting  acknowledg¬ 
ment  simultaneously.  Buffer  snace  is  necessary  for  some  32  pack¬ 
ets  assuming  a  50Kbs  channel  and  a  one  quarter-second  propagation 
up  to  the  synchronous  satellite  and  back  down.  That  is,  32  pack¬ 
ets  can  be  sent  out  before  the  acknowledgment  returns  for  the 
first.  The  next  hardware  addition  is  a  mechanism  fc  signaling 
the  satellite  radio  transmitter  when  to  turn  the  radio  carrier 
on  and  off  as  packets  are  transmitted;  this  is  necessary  because 
if  two  satellite  ground  stations  have  their  radio  transmitter 
carriers  on  simultaneously,  they  jam  each  other.  The  third 
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addition  is  time-keeping  hardware  necessary  for  the  Satellite 
IMPs  to  accomplish  accurate  slotting.  This  hardware  notes  the 
arrival  time  of  the  leading  edge  of  a  packet  (slot)  and  makes 
this  time  available  to  the  program  when  the  program  fields  the 
received  packet  interrupt  and  updates  the  program's  estimate  of 
the  slot  positions.  This  hardware  also  allows  the  program  to 
accurately  specify  transmission  of  a  packet  at  a  specified  time 
in  the  future. 

One  hardware  modification  was  necessary  for  construction  of 
the  Satellite  IMP.  The  IMP  modem  interface  normally  uses  a  unique 
character  sequence  to  denote  the  end  of  a  packet,  thus  requiring 
an  escape  character  with  escape-character  doubling  fox*  data  trans¬ 
parency.  Escape-character  doubling,  however,  can  result  in  the 
length  of  a  packet  being  temporarily  increased  while  traversing 
the  satellite,  thus  overflowing  a  slot.  The  IMP  modem  interface, 
therefore,  had  to  be  modified  to  use  a  word  count  to  specify 
packet  length. 

5.2  Software 

A  number  of  software  changes  and  additions  are  necessary  to 
convert  the  normal  IMP  into  a  Satellite  IMP. 

We  have  implemented  a  slotting  algorithm  which  operates  as 
follows:  each  Satellite  IMP  tracks  the  location  of  the  leading 

edges  of  all  packets  transmitted  by  other  Satellite  IMPs  and 
averages  them  with  exponential  weighting  to  determine  the  average 
slot  position,  which  is  used  as  the  standard. 

We  have  added  a  mechanism  for  randomizing  retransmissions. 
Whenever  there  is  a  packet  for  retransmission,  we  simply  transm.'t 
into  a  slot  or  not,  with  a  constant  probability  for  each  slot. 
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This  is  very  simple  to  implement  and  nearly  equivalent  in  effect 
to  the  harder-to-implement  retransmission  scheme  analyzed  exten¬ 
sively  by  Kleinrock  and  Lam  [55  8,  9]. 

The  IMP  had  to  be  modified  to  process  routing  data  coming 
from  several  different  sources  over  a  single  channel.  Each  IMP 
can  only  be  allowed  to  sene',  routing  data  over  the  satellite 
channel  once  each  routing  oeried  even  though  an  IMP  normally 
sends  routing  to  each  of  its  neighbors  each  routing  period.  When 
the  IMP  decides  to  route  a  packet  out  the  satellite  circuit,  it 
must  declare  which  Satellite  IMP  at  the  other  end  of  the  circuit 
is  to  receive  and  forward  the  packet  to  its  destination.  Neigh¬ 
boring  IMPs  normally  exchange  a  pair  of  messages  at  least  once 
each  routing  period  to  determine  whether  the  IMP  at  the  other 
end  of  the  circuit  (or  the  line  between  them)  is  alive.  Over 
the  broadcast  satellite  circuit,  rather  than  pairwise  exchange 
of  these  messages,  each  Satellite  IMP  broadcasts  a  message  stating 
which  of  the  other  IMPs  it  has  heard  from  (and  it  thus  considers 
alive)  since  the  last  period. 

The  inter-IMP  acknowledgment  protocol  normally  allows  only 
eight  packets  to  be  outstanding,  awaiting  acknowledgment,  at  a 
time.  The  satellite  channel  required  expansion  to  thirty-two 
packers  outstanding  at  a  time.  To  simplify  and  optimize  the 
inter-IMP  acknowledgment  scheme  between  Satellite  IMPs,  packets 
are  acknowledged  by  the  slot  in  which  the  packet  arrived  rather 
than  independently  for  each  source  IMP. 

Neighboring  IMPs  normally  calculate  the  unused  capacity  and 
the  delay  over  the  circuit  between  them.  This  information  is 
important  to  the  IMPs'  routing  computation  and  the  propagation  of 
routing  information  across  the  network.  The  satellite  version  of 
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the  IMP  will  eventually  have  to  take  Into  account  that  the  delay 
over  the  broadcast  satellite  channel  is  statistical  rather  than 
fixed  and  that  the  excess  capacity  may  have  different  character¬ 
istics  over  the  broadcast  satellite  channel.  The  statistical 
nature  of  the  delay  over  the  satellite  channel  will  undoubtedly 
have  important  consequences  for  network-wide  timing. 


Finally,  use  of  a  slotted  channel  requires  modification  to 
the  IMP  program  reloading  mechanism.  Previously,  IMPs  reloaded 
by  requesting  and  receiving  a  complete  core  image  in  a  single 
transmission  from  an  adjacent  IMP.  This  technique  has  to  be 
modified  to  send  the  core  image  one  packet  at  a  time. 

5.3  Characteristics 


The  Satellite  IMP  as  presently  constructed  includes  all  the 
features  of  a  normal  IMP  including  the  ability  to  have  Host  con¬ 
nections.  The  Satellite  IMP  can  have  multiple  ground  circuits 
(fewer  than  five)  to  the  terrestrial  network,  but  only  one  packet 
broadcast  satellite  circuit.  The  Satellite  IMP  presently  under 
construction  will  allow  a  satellite  channel  to  be  shared  with 
only  a  small  number  of  other  Satellite  IMPs.  While  the  initial 
machine  will  undoubtedly  be  used  with  a  pOKbs  satellite  channel, 
it  has  the  capacity  to  handle  a  channel  on  the  order  of  200Kbs. 

In  fact,  early  calculations*  suggest  that  Satellite  IMP  perfor¬ 
mance  will  have  an  upper  bound  of 

7c/n  +  c  <  1200  Kbs 

where  n  is  the  number  of  Satellite  IMPs  sharing  the  channel  and 
c  is  the  effective  channel  capa^Jty.  As  with  the  IMP  system,  the 
Satellite  IMP  system  will  ultimately  have  some  capability  for 
measuring  the  performance  and  behavior  of  the  Satellite  IMP  and 
broadcast  channel. 

#QTR  No.  ITOctober  1973. 
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5.4  Testing 

Testing  the  Satellite  IMP  requires  a  satellite  channel  simu¬ 
lator.  For  the  purpose  of  testing  a  random  ALOHA  system,  a 
satellite  simulator  was  easily  provided  by  inserting  a  conven¬ 
tional  IMP  with  special  software  between  two  Satellite  IMPs .  The 
IMP  is  used  as  a  quarter-of-a-second  delay  line  between  the 
Satellite  IMPs,  destroying  packets  from  the  two  Satellite  IMPs 
which  are  transmitted  simultaneously.  A  more  complete  simulation 
is  required  to  test  a  version  of  the  system  using  slotting.  Phis 
we  provided  by  building  a  small  hardware  box  to  which  four  Satel¬ 
lite  IMPs  can  be  connected.  The  box  ORs  all  data  from  the  Satel¬ 
lite  IMPs  together,  runs  it  through  a  quarter-second  delay  line, 
and  passes  the  data  to  all  the  Satellite  IMPs.  Additionally,  if 
more  than  one  Satellite  IMP  has  its  carrier  on/off  signal  on  at 
any  time,  the  box  causes  the  two  or  more  Satellite  IMPs  to  jam 
ec  ch  other . 

5.5  Status 

The  most  recent  version  of  the  Satellite  IMP  follows  a  ran¬ 
dom  ALOHA  protocol  between  a  pair  of  nodes  (two  nodes  are  suf¬ 
ficient  to  test  a  broadcast  protocol).  V.'c  have  almost  completed 
another  version  of  the  Satellite  IMP  which  follows  a  slotted 
ALOHA  protocol  between  more  than  two  nodes.  A  version  of  the 
Satellite  IMP  following  a  conventional  round-robin  TDMA  protocol 
would  be  very  simple  to  implement,  and  we  may  experiment  with 
this  at  some  time  In  the  future.  Also,  we  will  undoubtedly  ex¬ 
periment  with  some  novel  reservation  protocol(s),  of  which  [3] 
and  [7]  have  been  given  the  most  analysis  to  date. 
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As  yet  no  Satellite  IMP  has  been  installed  in  the  ARPA 
Network,  mainly  because  of  difficulty  in  choosing  appropriate 
sites  and  obtaining  tariff  approval  to  carry  out  an  experiment 
in  packet  broadcast  communications.  Presently  we  expect  to 
install  the  first  two  Satellite  IMPs  in  the  first  half  of  197^. 
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